Highly probable identification of related messages using sparse hash function sets

ABSTRACT

Methods, systems, and apparatus for network monitoring and analytics are disclosed. The methods, systems, and apparatus for network monitoring and analytics perform highly probable identification of related messages using one or more sparse hash function sets. Highly probable identification of related messages enables a network monitoring and analytics system to trace the trajectory of a message traversing the network and measure the delay for the message between observation points. The sparse hash function value, or identity, enables a network monitoring and analytics system to identify the transit path, transit time, entry point, exit point, and/or other information about individual packets and to identify bottlenecks, broken paths, lost data, and other network analytics by aggregating individual message data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to and claims priority to provisionalpatent applications Attorney Docket Nos. 130150P, now U.S. Patent Appl.No. 61/879,186, entitled “HIGHLY PROBABLE IDENTIFICATION OF RELATEDMESSAGES USING SPARSE HASH FUNCTION SETS”, 130258P, now U.S. PatentAppl. No. 61/879,188, entitled “METADATA CORRELATION ANDDISAMBIGUATION”, 130259P, now U.S. Patent Appl. No. 61/879,192, entitled“EXPLORATIVE VISUALIZATION OF COMPLEX NETWORKS IN CONSTRAINED SPACES”,the disclosure of each of which is incorporated by reference herein intheir entireties.

INTRODUCTION

Wired and wireless networks comprise complex graphs with non-trivialtopological features. Patterns of connections between network elements,as well as the state of the elements, are neither purely random norpurely regular. In addition, the complexity increases drastically whennetworks of networks are introduced. For organizations withmission-critical networks, for example, mobile networks or financialtrading networks, the inability to quickly and accurately assess thequality or performance of their respective networks, quickly determinethe root causes of challenges, and intelligently optimizeinfrastructure, costs the organizations millions of dollars.

Network and datacenter service providers have a fundamental need toprovide fast, reliable network services to their customers. Network anddatacenter services typically must meet or exceed an agreed upon levelof service quality, which may be defined in one or more Service LevelAgreements (SLAs). The network and datacenter service providers arerequired to provide proof that their services are meeting the terms ofthe SLA in the form of key performance indicators (KPIs). SLAs and KPIstypically include, but are not limited to, specific requirements forconnectivity, delay, jitter, throughput, uptime, mean time to repair(MTTR), and mean opinion score (MOS).

Current systems and methods are inadequate for providing comprehensivemonitoring and analytics of KPIs. Current techniques include monitoringequipment or software coupled to the network, manual or automatednetwork audits, or sampling of a short time period, a small portion of anetwork, or a small portion of the traffic in a network. Priortechniques are not scalable and are not equipped to respond as networksadd additional elements, become distributed or virtualized, and cannotrespond to demand increases for greater bandwidth with real-timelatency.

Traditional methods of visualizing networks are inadequate for providingreal-time monitoring and analytics of networks. Traditional methods,such as heat maps and other hierarchical displays, require large amountsof space to adequately display a network. If the visualization isconstrained to a small amount of space, such as a computer monitor, thenonly a small portion of the visualization can be displayed and the userloses critical insight due to a loss of information.

SUMMARY

In various embodiments, computer-implemented methods and systems aredisclosed. In one embodiment, the computer-implemented method comprisescalculating, by a processor, a hash value for a first message at aplurality of observation points on a network using a first hashfunction. The hash value is calculated for the invariant fields of themessage. The computer-implemented method further comprises associating,by the processor, metadata with the hash value of the first message;tracking, by the processor, the transit of the first message over thenetwork; and generating, by the processor, one or more network analyticsfor the first message over the network. The one or more networkanalytics are generated from the associated metadata.

In various embodiments, computer-implemented methods and systems aredisclosed. In one embodiment, a computer-implemented method comprisesreceiving, by a processor, a plurality of metadata packets correspondingto a plurality of messages. Each of the metadata packets comprises asparse hash value. The computer-implemented method further comprisesidentifying, by the processor, a plurality of matching sparse hashvalues. The plurality of matching sparse hash values correspond to atleast a first message and a second message. The computer-implementedmethod further comprises disambiguating, by the processor, the firstmessage and the second message. The first message and the second messageare disambiguated using the metadata associated with the plurality ofsparse hash values.

FIGURES

The novel features of the embodiments described herein are set forthwith particularity in the appended claims. The embodiments, however,both as to organization and methods of operation may be betterunderstood by reference to the following description, taken inconjunction with the accompanying drawings as follows.

FIG. 1 illustrates one embodiment of a network topology.

FIG. 2 illustrates one embodiment of a network topology comprising anintegrated network monitoring and analytics system.

FIG. 3 illustrates one embodiment of an IPv4 packet during transit of anetwork.

FIG. 4 illustrates one embodiment of an IPv4 packet during transit of anetwork comprising an invariant source address.

FIG. 5 illustrates one embodiment of sparse field handing of a packet bya hash function.

FIG. 6 illustrates one embodiment of derived metadata for an IPv4packet.

FIG. 7 illustrates one embodiment of metadata collected at variousobservation points within a network.

FIG. 8 illustrates one embodiment of discrete time periods used fordisambiguation of hash function values.

FIG. 9 illustrates one embodiment of incremental aggregation ofmetadata.

FIG. 10 illustrates one embodiment of a network visualization display.

FIG. 11 illustrates a section of the network visualization display ofFIG. 10.

FIG. 12 illustrates one embodiment of a chord diagram comprising aplurality of chords.

FIG. 13 illustrates one embodiment of a computing device which can beused in one embodiment of the systems and methods for network monitoringand analytics

DESCRIPTION

Reference will now be made in detail to several embodiments, includingembodiments showing example implementations of systems and methods fornetwork monitoring and analytics. Wherever practicable similar or likereference numbers may be used in the figures and may indicate similar orlike functionality. The figures depict example embodiments of thedisclosed systems and/or methods of use for purposes of illustrationonly. One skilled in the art will readily recognize from the followingdescription that alternative example embodiments of the structures andmethods illustrated herein may be employed without departing from theprinciples described herein.

In various embodiments, methods, systems, and apparatus for networkmonitoring and analytics are disclosed. In some embodiments, the systemsand methods for network monitoring and analytics comprise highlyprobable identification of related messages using one or more sparsehash function sets. In some embodiments, the systems and methods fornetwork monitoring and analytics comprise metadata correlation anddisambiguation. In some embodiments, a network visualization display isdisclosed.

In various embodiments, systems and methods for network monitoring andanalytics are disclosed. The network monitoring and analytics system isscalable to monitor and track the transit of up to every message andevery node across a distributed network, for example, a nationwidedistributed network. The network monitoring and analytics system isfurther scalable to monitor every message communication process in,among, and between virtual and/or physical servers, switches, androuters in, among, and between datacenters. Statistics are collectedbased on the message transits including, but not limited to, messagesize, message type, message source(s) and/or originator(s),destination(s), delay per observation point pair, loss, locationsbetween which the loss occurred, and transit topology.

In some embodiments, the systems and methods for network monitoring andanalytics perform highly probable identification of related messagesusing one or more sparse hash function sets. Highly probableidentification of related messages enables a network monitoring andanalytics system to trace the trajectory of a message traversing thenetwork and measure the delay for the message between observationpoints. Statistics are developed for individual messages and/orensembles of messages. The sparse hash function value, or identity,enables a network monitoring and analytics system to identify thetransit path, transit time, entry point, exit point, and/or otherinformation about individual packets and to identify bottlenecks, brokenpaths, lost data, and other network analytics by aggregating individualmessage data. In various embodiments, the sparse hash function iscalculated for one or more invariant fields of a message. The invariantfields used for the hash function calculation may depend, for example,on the protocol, message type, network type, and/or other parameters ofthe message and/or network.

In some embodiments, the systems and methods for network monitoring andanalytics comprise metadata collection. The collected metadata is usedfor network analysis and hash value disambiguation. The metadatacollected for a message is associated with the message within thenetwork. The metadata is used to disambiguate two or more messagescomprising the same hash value.

In some embodiments, the systems and methods for network monitoring andanalytics comprise a network visualization display. The networkvisualization display provides visualization of network states, flows,and relationships between nodes of a network. In some embodiments, thenetwork visualization display comprises a chord diagram. The chorddiagram comprises a plurality of nodes and one or more chords connectingat least a subset of the plurality of nodes. The network visualizationdisplay provides network operators with real-time, easy to interpretinformation regarding network utilization and functionality.

In various embodiments, the network monitoring and analytics systems andmethods are configured to provide visual indicators that allow a userto, at-a-glance, quickly understand large-scale, distributed and complexnetworks, including the states of the network elements, the flowsbetween the network elements, the relationships amongst the networkelements, and the problems that are occurring in real-time.Mission-critical networks can scale to hundreds of terabits per second.In various embodiments, the network monitoring and analytics systemcomprises a visual display. The visual display is configured to providevisualization of network statistics in a format configured for fastidentification of flows, errors, and/or other network information. Insome embodiments, the visual display comprises a chord diagram.

In various embodiments, a message comprises a collection of bits and/orbytes that represent information that is to be transported between twoor more locations. A message may be a file, a portion of a file, aprotocol message, and/or any other type of message on the network, thatis stored on a storage medium and/or that is in flight (in transit) in anetwork. In some embodiments, a message comprises a service data unit(SDU), a protocol data unit (PDU), a datagram, a packet, a frame and/ora cell. Those skilled in the art will recognize that a message maycomprise any digital bits and/or bytes transmitted over a network and isnot limited to only those messages and/or protocols discussed herein. Amessage may originate on, terminate on, and/or pass through the network.

When transmitted over a network, a message may be transmitted withinand/or as a single packet (including, but not limited to, the networkstandard definitions of packet, frame or cell), within a fraction of apacket, distributed across multiple packets that may traverse differentpaths, and/or may be fragmented into multiple packets. Messages may beunicast, multicast or broadcast to a number of destinations andpotentially replicated by network elements. A packet may be encapsulatedin one or more additional packets generated during transit of themessage over the network.

Sparse Hash Function Calculation

FIG. 1 illustrates one embodiment of a network topology 102. The networktopology 102 comprises a mobile service provider network. Although amobile service provider network is illustrated, the network topology 102is provided merely as an illustration, and is not intended to belimiting. Those skilled in the art will recognize that the networktopology 102 is illustrative of a typical network and the sameprinciples will apply to other network topologies. The network 102comprises a plurality of user devices 104 a-104 d. The plurality of userdevices 104 a-104 d are configured to transmit and/or receive messagesover the network 102.

In one embodiment, a user device 104 a generates a message to betransmitted over the network 102. The user device 104 a transmits themessage to a base station 106 a, 106 b coupled to the network 102.Although a wireless transmission is illustrated, the user devices 104a-104 d may be coupled to the service provider network 102 through wiredand/or wireless connections. The base station 106 a, 106 b receives themessage and retransmits the message to a backhaul network 108 a, 108 b.The time between the message being received at the base station 106 a,106 b to the time that the message is transmitted from the base station106 a, 106 b to the backhaul network 108 a, 108 b comprises a firsttransit time d1. The message traverses the backhaul network 108 a, 108 band is transmitted from the backhaul network 108 a, 108 b to one or morenetwork components. For example, in one embodiment, the message istransmitted to a radio network controller (RNC) 110. The time betweenthe message being received by the backhaul network 108 a and beingreceived at the network component comprises a second transit time d2.

A network component may process, transform, and/or otherwise alter themessage and transmit the message to one or more additional networkcomponents. For example, the RNC 110 processes the message and transmitsthe message to a serving GPRS (general packet radio service) supportnode (SGSN) 112. The time between receipt of the message at the RNC 110and receipt of the message at the SGSN 112 comprises a third transittime d3. The SGSN 112 processes the message and transmits the message toa gateway GPRS support node (GGSN) 114. The time between receipt of themessage at the SGSN 112 and receipt of the message at the GGSN 114comprises a fourth transit time d4. The GGSN processes the message andtransmits the message to a network outside of the mobile serviceprovider network 102, for example, to the internet 116. The time betweenreceipt of the message at the GGSN 114 and the message exits the networkcomprises a fifth transit time d5. The sum of the transit times d1, d2,d3, d4, and d5 comprises the total transit time for the message on thenetwork 102.

In some embodiments, one or more network components may be substitutedfor one or more other network components. For example, the RNC 110, theSGSN 112, and the GGSN 114 may be replaced by a single servinggateway/PDN gateway (SPGW) 118. The SPGW 118 may process the message andperform each of the functions of the RNC 110, SGSN 112, and GGSN 114 ina single device. A single transit time may be calculated from the time amessage is transmitted to the SPGW 118 to the time the SPGW 118transmits the message outside of the network 102. In some embodiments,one or more of the network components, such as, for example, the RNC110, the SGSN 112, the GGSN 114, or the SPGW 118 modify a message and/ora packet encapsulating the message. Network owners may be interested inthe individual transit times between network elements, the aggregatetransit time of a message over the network, and/or additional messageinformation.

FIG. 2 illustrates one embodiment of a real-time network performancemonitoring system 200. The real-time network performance monitoringsystem 200 comprises a network 202. The network 202 may comprise, forexample, a mobile provider network, a datacenter provider network,and/or any other network. The network 202 comprises a plurality of userdevices 204 a, 204 b. The plurality of user devices 204 a, 204 b areconfigured to transmit and/or receive messages over the network 202. Theplurality of user devices 204 a, 204 b are coupled to a Base TransceiverStation evolved Node B (BTS eNodeB) 206. In one embodiment, a message istransmitted from a user device 204 a to the BTS eNodeB 206. The BTSeNodeB 206 is coupled to one or more cell site routers (CSRs) 208 a-208d. An aggregation router 210 couples the CSRs 208 a-208 d to a metroarea network (MAN). The MAN couples the CSRs 208-208 d to a core router212, a SPGW 214, and a wide-area network (WAN) router 216. The WANrouter 216 couples the local network 202 to one or more additionalnetworks, such as, for example, the internet 218 and/or a backbone WAN220.

A network monitoring and analytics system 222 is integrated with thenetwork 202. The network monitoring and analytics system 222 isconfigured to monitor messages traversing the network 202 and todetermine transit times, transit paths, source, destination, length,and/or other network analytic information for each message traversingthe network. The network monitoring and analytics system 222 generates asparse hash function value for each message traversing the network. Thespare hash function value provides highly probable identification ofrelated messages. The network monitoring and analytics system 222comprises a plurality of observation points 224 a-224 e. Eachobservation point 224 a-224 e is monitored and the sparse hash functionvalue for each of the messages passing through the observation points224 a-224 e is recorded. Network analytics data is generated bycomparing message information at each of the observation points 224a-224 e. In some embodiments, the network monitoring and analyticssystem 222 provides end-to-end and/or node-to-node monitoring of delay,jitter, throughput, and loss. The network monitoring and analyticssystem 222 is embedded into the network elements and user equipment andeliminates the need for additional, external probes.

In some embodiments, the network monitoring and analytics system 222generates metadata for up to every packet at every observation point 224a-224 e within the network 202. The metadata may be monitored, filtered,and provided to an analytics engine of the network monitoring andanalytics system 222. The metadata is aggregated, correlated, andanalyzed by the network monitoring and analytics system 222, forexample, by a Mobile Switch Center (MSC). In some embodiments, thegenerated metadata is used for disambiguation of messages having thesame sparse hash function value.

The network monitoring and analytics system 222 provides link, segment,and path statistics from each of the user devices 204 a, 204 b to thenetwork core and provides upstream, downstream, one-way, and/orround-trip analysis. A heat map may be generated for visualization ofthe network 202 and/or the messages traversing the network 202. In someembodiments, the network monitoring and analytics system 222 providesthreshold crossing alerts and allows flexible queries to analyzestatistics per node, hop, equipment type, geography, protocol,subscriber, time of day, and/or other parameters. In some embodiments,the network monitoring and analytics engine 222 provides wire speedprocessing rates of up to, for example, 2×100 GbE per analyticsappliance with an effective traffic analysis of, for example, 800 Gbps.

In some embodiments, the network monitoring and analytics system 222utilizes sparse hash function sets to generate unique identifiers foreach packet and/or message within a network 202. A hash functioncomprises an algorithm that maps data of variable length, for example, amessage, to data of a fixed length. In some embodiments, the hashfunction value is smaller than the message. A message typically containsstatic, or invariant, information that is to be transported over thenetwork and dynamic, or variant, information that is changed, ormodified, as the message traverses the network. The portions of themessage that change comprise variant fields and the portions of themessage that do not change comprise invariant fields. One or morevariant fields may be modified in well-known ways as the message istransported in the network. In some embodiments, the one or more variantfields modified in well-known ways are mapped to an invariant value thatis used in the hash function. For example, in one embodiment, thewell-known variant fields are mapped to a constant value in the hashfunction.

Fields may comprise individual bits and/or collections of bits, such as,for example, eight bits (a byte). Messages may comprise fixed orvariable length, in that the length of the message itself may bemodified as the packet traverses the network. Different messages maycomprise fixed and/or variable lengths. For example, one message maycomprise a fixed length of 64 bytes and another message may comprise afixed length of 1044 bytes, or any length allowed on the network or inthe definition of the message. The length of the message may vary, forexample, depending on the protocol used to transport the message overthe network 202.

In some embodiments, the sparse hash function may be implemented inhardware, software, or a combination thereof. For example, in variousembodiments, the sparse hash function may be implemented in hardware asan application specific integrated circuit (ASIC), a field programmablegate array (FPGA), and/or other specialized hardware. In someembodiments a sparse hash function may be implemented in softwareconfigured to be executed by a central processing unit (CPU), graphicalprocessing unit (GPU), or other general purpose processor. In someembodiments, the sparse hash function may comprise both hardware andsoftware portions.

In various embodiments, the sparse hash function provides a uniqueidentity, or thumbprint, for each packet that enters the network. Forexample, in one embodiment, a network may comprise four observationpoints. A packet enters the network at a first observation point, or afirst node. The sparse hash function generates a hash value for thepacket and associates the generated hash value with, for example,metadata associated with the packet. As the packet traverses thenetwork, the packet is identifiable at each node by the hash valueidentity generated for the packet. For example, if the packet travelsfrom the first node to a second node, the hash value for the packet iscalculated at the second node and identifies the packet as the packetthat entered the network at the first node. By identifying the packet ateach node, the path of the packet through the network can be identifiedby aggregating each node through which the packet passes. In variousembodiments, the sparse hash function allows the network monitoring andanalytics system to monitor up to every packet passing through adistributed network.

In various embodiments, metadata may be associated with the hash valuegenerated for a packet. Packet metadata comprises one or more fieldsconfigured to provide a description of the packet. Packet metadata isused to perform network analysis and aggregation without needing totransmit or analyze an entire packet. In some embodiments, the hashvalue and metadata for each packet is provided to a network analyticssystem. The network analytics system identifies packets comprisingmatching hash values and analyzes the packet's trajectory through thenetwork based on metadata associated with the hash value. Associatedmetadata may comprise, for example, the size of the packet, time oftransmission, trajectory through the network, the source of the packet,the destination of the packet, type of packet, and/or additional packetmetadata. In some embodiments, the metadata information stored for eachpacket comprises a fraction of the size of the packet. By providing acompact representation of each packet on the network, the metadata andsparse hash function allow monitoring and analysis of up to every packeton a network.

FIG. 3 illustrates a message 302 a sent as an Internet Protocol version4 (IPv4) packet transported over an IEEE 802.2 Ethernet frame. Althoughan IPv4 packet is illustrated, those skilled in the art will recognizethat any protocol, frame, cell, and/or combination thereof may be usedwith the present disclosure. The message 302 a is observed at fourobservation points as it traverses a network. The original message 302 acomprises a plurality of variant and invariant fields. The variantfields of the message 302 a comprise, for example, a media accesscontrol (MAC) Destination Address 306 a, a MAC Source Address 310 a, anether type 312 a, a packet identification 316 a, one or more flags 318a, a header checksum 322 a, a source address 324 a, and/or a frame checksequence 326 a. In embodiments including protocols other than IPv4, thevariant fields of the message may include additional, fewer, and/oralternative variant fields. The original message 302 a comprises aplurality of invariant fields 308 a-308 f, such as, for example, aversion length field 308 a, a differentiated services codepoint/explicit congestion notification (DSCP/ECN) field 308 b, a totallength field 308 c, a protocol field 308 d, a destination address field308 e, and/or a payload field 308 f. In embodiments including protocolsother than IPv4, the invariant fields of the message may includeadditional, fewer, and/or alternative invariant fields.

As the message 302 a (or packet) traverses the network, the message 302a may be transformed by one or more network components and/or protocols.For example, in the illustrated embodiment, the message 302 a istransformed by Network Address Translation (NAT), entry to a GeneralPacket Radio Service (GPRS) Tunneling Protocol (GTP) tunnel, exit fromthe GTP tunnel, and subsequent handoff to the internet. The originatingmessage 302 a is shown on the left. Upon entry to the GTP Tunnel, themessage 302 a is tagged with an IEEE 802.1Q Virtual Local Area Network(VLAN) tag and encapsulated within an IPv4 packet 302 b as a UserDatagram Protocol (UDP) datagram that is a GTP message containing theoriginal message 302 a. Encapsulation of the message within an IPv4packet adds additional variant fields 328 to the message 302 b. Somefields of the original message 302 a are changed due to NAT includingthe original message's IP Source Address 324 b, identification 316 a,flags 318 a, Time To Live (TTL) field 320 b and IP Header Checksum 326b. The Ethernet MAC Source 310 a-310 d and Destination Addresses 306a-306 d are changed by the traversal of routers and the Frame CheckSequence (FCS) has been recalculated.

Despite the many modifications to variant fields of the original message302 a, the invariant fields 308 a-308 f remain unchanged. The invariantfields 308 a-308 f are shown as unshaded in the four packets 302 a-302d. As the message 302 a transits routers and traverses the path of theGTP tunnel, the Ethernet MAC Source 310 b, 310 c and Destinationaddresses 306 b, 306 c and FCS 326 b, 326 c are changed as is theEncapsulating IPv4 TTL 320 b and Header Checksum 322 b. The invariantfields of the original message 302 a are not modified within the GTPtunnel. When the message 320 c exits the GTP tunnel, the message 320 cis again transformed for its delivery to the internet by removing theGTP, UDP and IPV4 encapsulations and VLAN tag. The Ethernet MAC SourceAddress 310 c, Destination Address 306 c and FCS 326 c of the message302 b, 302 c are changed as appropriate. The original message's 302 aTTL 320 a is decremented and its Header Checksum 326 a is recomputed.The fields that are unshaded in the final packet 302 d are those fieldsthat are invariant across the entire transit path. In one embodiment,the invariant fields, or a subset thereof, are utilized to calculate asparse hash function value for the message.

In various embodiments, the invariant fields 308 a-308 f are defined bythe network protocols that act on the message at the various switchesand routers along the message's path. In the illustrated embodiment, thefields are shown with the width representing one byte and the heightrepresenting the number of bytes. In other embodiments, fields may beany number of bits and/or collections of bits and need not be contiguousnor adjacent and are not constrained to integer byte sizes. Forinstance, fields, or sub-fields, in the IPV4 packet comprise 4 bits(Version, IP Header Length (IHL)), 3 bits (Explicit CongestionNotification (ECN), Flags), 5 bits (Differentiated Services Code Point(DSCP)) and 13 bits (Fragment Offset). Encapsulations may vary in size.For example, IPv4 supports an optional field of up to 40 additionalbytes in the packet header.

Traversal through NAT can modify a number of fields of the originalmessage 302 a and is protocol dependent. For instance, the originalmessage 302 a comprises a UDP datagram. The original message's UDPsource port and UDP Header Checksum may be modified by NAT. Otherprotocols, such as, for example, TCP or ICMP, have different headerlengths and different fields that may be modified by NAT. Certainapplications and/or protocols, such as, for example, File TransferProtocol (FTP), comprise fields that may not be encoded as fixed lengthbinary, but as variable length character strings encoded as AmericanStandard Code for Information Interchange (ASCII) bytes. The charactersof the strings, as well as the length of the strings, may be changed byNAT. The location of one or more fields within a packet may be moved byNAT, including variant and invariant fields.

In some embodiments, a packet may be fragmented as the packet transitsthe network. If a first fragment is greater than the size of the maximalsize of the fields of the packet that are used for the hash, the hash iscomputed over the first fragment. If a fragmented packet is partially orwholly less than the number of bytes used for the hash function, aplurality of fragments are used until the hash is complete. Fragmentsmay be discarded if the end or intermediate fragments are missing and/ordo not arrive within a maximal accepted time.

In some embodiments, the network monitoring and analytics systemcomprises a functional block. The functional block determines theexistence and location of the original fields of message. The functionalblock may utilize knowledge of the functional blocks location and/oralgorithmic calculations to determine the existence and location of theoriginal fields of the message. For example, in one embodiment, a packetis scanned for a VLAN tag, IPV4, UDP and GTP encapsulations returning anoffset pointing to the original message. In embodiments comprisingvariable sized transformations, the function returns a list of pointersand lengths to the various invariant fields and/or block of fields. Inmany networks, packets entering the network from the outside, forexample, the internet, will experience different transformations fromthose egressing the network. FIG. 4 illustrates the invariant fields fora packet 402 a entering the network. For a packet 402 a entering thenetwork, the IPv4 source address 408 e is invariant and the IPv4Destination 424 a is variant. In contrast, the invariant fields for apacket 302 a entering the network comprise an invariant IPv4 destinationaddress 308 e and a variant IPv4 source address 324 a, as illustrated inFIG. 3. A packet 402 a entering the network undergoes similar processingand transformation as those discussed with respect to FIG. 3.

In some embodiments, the sparse hash function computes a hash over asubset of the invariant fields 308 a-308 f, 408 a-408 h, that are, inturn, a subset of the original message 302 a, 402 a. The choice of thesubset is protocol (and application) dependent as determined by scanningof the packet and/or message. The sparse hash function acts upon onlythe invariant fields of the packet by eliminating (deleting or makingconstant) the various variant fields and may be limited, for example, toselect only a subset of the invariant fields. The subset of theinvariant fields selected may be protocol (and application) dependent.One or more fields may be excluded and/or limited to a certain number ofbytes. The one or more fields that are excluded and/or ignored, andthose which are utilized by the hash function, need not be contiguous.

The sparse hash function generates a hash value that is highly probableto be unique for the message that is hashed. It is not necessary thatthe entire message be hashed, for example, if the messages are likely tobe of different lengths and/or contents. Generation of message hashesthat are highly likely to be unique and that can be disambiguated bylength, type, address, location, time frame, and/or other parameters,ensures highly probable identification of packets. Hash functions aretypically costly in terms of gates (if implemented in hardware) and CPUcycles (if implemented in software). Reduction of the number of bytes inthe hash results in a greater processing rate (messages hashed persecond) and a reduction in the power required for the hash per message,allowing the network monitoring and analytics systems and methods toobserve up to every message traversing a network.

FIG. 5 illustrates one embodiment of the sparse hash function handlingof one or more fields of a message 502 a. The original packet 502 a isshown on the left. The original packet 502 a comprises a plurality ofinvariant fields 506 a-506 i and variant fields 508 a-508 c, 510 a-510g. A sparse packet 502 b comprises the original packet 502 a as treatedby the hash function, as shown on the right. The sparse packet 502 bcomprises the invariant fields 506 a-506 i of the original packet 502 a.The invariant fields 506 a-506 g are generally unaltered in the sparsepacket 502 b. In some embodiments one or more invariant fields 506 i maybe truncated and part of the field ignored. A first subset of thevariant fields 508 a-508 c are ignored by the hash function. A secondsubset of the variant fields 510 a-510 g are set to a constant value bythe hash function.

In the illustrated embodiment, fields in the “sparse” packet that areshaded are ignored and fields that are unshaded are invariant fields 506a-506 i and/or the second subset of variant fields 510 a-510 g that havebeen set to a constant value. For example, in the illustratedembodiment, each of the second subset of variant fields 510 a-510 g hasbeen set to zero by the hash function. In some embodiments, setting thefields to zero as opposed to removing or ignoring the fields providesoptimal memory and computation costs. One or more fields 508 a, 508 c ateither end of the message 502 b may be ignored for the computation andmemory savings. The choice of ignoring or setting a field to a constantvalue is implementation dependent.

In some embodiments, the hash function is highly probable to be uniqueover the set of messages for which a hash is computed, is small enoughin a number of bits so as to be compactly stored or communicated, and iseasy to compute in hardware and/or software. There is a tradeoff betweenthat uniqueness of the hash value and the length of the hash due to theknown limitation of hash functions. For example, as the number ofmessages for which the hash is calculated increases, the chance that twounrelated messages will generate the same hash value also increase. Thechance that two unrelated messages will generate the same hash value ishigher for shorter hashes. This problem is referred to as the “birthdayproblem.” When two different messages result in the same hash value, acollision occurs. The probability of collisions is dependent on the hashfunction and extremely sensitive to the number of messages to be hashed.In general, a longer hash will have fewer collisions for the same numberof messages. But a longer hash is more costly to compute in terms ofspeed, gates, power, cycles and storage space. Whether the hash is 32bits, 64 bits, 128 bits, 256 bits or some other size, it is necessaryfor the hash value to be the same for a given message at all points thatare observed in order for that message to be identified as havingtraversed the observation points. In various embodiments, the hash maybe different for different message types, applications, and/or networks.For instance, in one embodiment, a 256 bit hash is used for a firstmessage type for which it is highly important that the hash values beunique over the message set and where there may be a very large numberof messages. A 32 bit hash is used for a second message type that has alower importance and/or a smaller number of expected messages.

In some embodiments, messages are associated with other messages for thepurpose of aggregate (ensemble) statistics. Messages can be grouped bytype and/or by classification. In some embodiments, messages are groupedtemporally into a flow. Flow statistics are of especial importance asthey are indicative of throughput and variation of delay, or jitter. Asmentioned above, SLAs routinely define the maximally acceptable jitterand thus there is a need to monitor flows and flow statistics. In oneembodiment, the network monitoring and analytics system enables tracingof a message's transit through a point or set of points within a networkto a high degree of confidence.

Metadata Correlation

As described above, in various embodiments, the hash value identifies amessage to a high degree of certainty. The hash value, or identity,further enables the identification of a message even if the message haspassed through one or more network devices that change some portion ofthe message and/or transport the message in a number of ways, such as,for example, via frames, packets, and/or datagrams with a potential forfragmentation. The identity of the message is used in combination withother information derived from the message, the message metadata, thelocation of the observation point and the time of the observation foranalysis of the network. The combination of information comprisesmetadata derived from the message. An example of metadata for an IPv4packet comprising a message is illustrated in FIG. 6. The exact numberof fields, their size and position within the metadata areimplementation dependent and may depend on, for example, the protocol,frame, cell, message length, network, and/or other factors.

FIG. 6 illustrates one embodiment of message metadata 628 collected foran IPv4 packet at a first observation point. The message metadata 628comprises metadata type 630 and metadata length 632 identifiers. Themessage metadata 628 further comprises observation point dependentmetadata, such as, for example, an observation point field 634 and atimestamp 638. The message metadata 628 further comprises messagespecific metadata, such as, for example, a message identity 636generated by the sparse hash function or a plurality of sparse hashfunctions, a message type 640, a message length 642, a message protocol644, a source address 646, a destination address 648, a source port 650,and a destination port 652.

In one embodiment, the metadata created at each observation point in themessage's trajectory is similar. Different observation points maygenerate more or less fields in the metadata. For example, asillustrated in FIG. 7, the metadata fields for a plurality ofobservation points 728 a-728 d may vary. For example, in the illustratedembodiment, each of the observations points 728 a-728 d collectmetadata, such as, for example, observation point identifiers 736 a-736d, a timestamp field 738 a-738 d, message type 740, and/or additionalmetadata. A second observation point 702 b and a third observation point702 c, corresponding to the entry and exit of the GTP tunnel, compriseadditional fields, including the 802.1Q VLAN tag 754, the IPV4 Source756 and Destination Addresses 758 of the tunnel, and the GTP TunnelEndpoint Identifier (TEID) 760. In some embodiments, metadata from twoor more observation points 728 a-728 d is associated, or correlated, bymatching the identities of the message at each observation point 728a-728 d. Although the hash-generated identities 734 are highly probableto be unique to a given message for a given total number of messages, itis still possible that identities may be the same for differentmessages. In some embodiments, the metadata fields are used todisambiguate different messages with matching hash generated identities.In various embodiments, the generated metadata is provided to a networkanalytics system and used for analysis of the network.

Disambiguation Of Hash Collisions

As discussed above, collisions may occur over a large message set. Thenetwork monitoring and analytics systems and methods are configured todisambiguate collisions. In various embodiments, the network monitoringand analytics system disambiguates messages through secondary, tertiary,or additional hashes (such as, for example, different hashes of the sameand/or different subsets of the invariant fields), the length of themessage, any or all subset fields of the original and/or transformedfields of the message, the location of the observation points, thetopology of the observation points and/or the time(s) of theobservation.

In some embodiments, message type fields source and/or destinationports, and/or length of the message are used for disambiguating themessages with matching hash identities 734. For instance, two messageswith the same hash identity but different message lengths are differentmessages, regardless of the matching hash identity. Other fields of thepacket such as IPv4 source, observation point, location, topology andobservation time(s), may be used to disambiguate one or more messages.

FIG. 8 illustrates one embodiment of disambiguation of two messages bytime. All other fields of the packet being the same, messages can bedisambiguated by excluding messages that fall outside of an “existence”window 862, 864. The existence window 862, 864 is relative to the timeof one message at a particular observation point, versus another messageat the same observation point.

For example, in the illustrated embodiment, Message X arrives at timeTarrival(X) 866. Tarrival(X) 866 is within a first time range 862.Message X's hash value, for the purposes of trajectory monitoring, onlyextends to include the earliest possible originating time of the packetto the latest possible time of egress to the network, as shown by thefirst time range 862. A second message, Message Y, arrives at timeTarrival(Y) 868. Message Y comprises the same hash value as Message X.Message X and Message Y may be disambiguated, as Tarrival(Y) 868 fallsoutside of the possible existence times for Message X, the first timerange 862, and within a possible existence time for Message Y, thesecond time range 864.

In some embodiments, messages comprising matching hash values aredisambiguated by location and/or trajectory. A path of a packetcomprising a first message is constrained to a first subset ofobservation points and a path of a packet comprising a second message isconstrained to a second subset of observation points. The first andsecond subsets of observation points are partially and/or whollynon-overlapping. Messages are disambiguated by observation point andtrajectory, as the first packet and the second packet comprisedifferent, unique paths and/or trajectories through the network.

Incremental Aggregation Of Metadata.

FIG. 9 illustrates one embodiment of incremental aggregation ofmetadata. Incremental aggregation of metadata can be performed asdesired at any point in the network. The metadata at each observationpoint within the network is retained for a period of time that allowsmetadata from other observation points to traverse to an aggregationpoint. Incremental aggregation increases the efficiency of the networkas the bandwidth for aggregated metadata is less than multiple separatemetadata transmissions. The aggregated metadata comprises multipleobservation point and timestamps, along with the various fieldsaggregated without duplication. The aggregation point generates anaggregated metadata set 928. The aggregated metadata set 928 comprisesinvariant metadata, such as, for example, a metadata type 930, ametadata length 932, a message identity 934, a message type 940, amessage length 942, a protocol 944, and/or a destination port 952. Theaggregated metadata set 928 further comprises observation pointdependent metadata, such as, for example, observation point identifiers936 a-936 c, timestamps 938 a-938 c, and/or metadata generated by theobservation point, such as, for example, an 802.1Q tag 954, a sourceaddress 956, a destination address 958, and/or a TEID tag 960.

Explorative Visualization of Complex Networks in Constrained Space

In some embodiments, the statistics generated by the network monitoringand analytics system are presented to a user as a visual output. Thevisual output provides monitoring of up to every message at every nodeacross a distributed network, such as, for example, a nationwidedistributed network and/or monitoring of up to every messagecommunication process in, among, and between virtual and/or physicalservers, switches, and routers in, among, and between datacenters. Atany given time, in a large-scale, distributed and complex network, therecan be tens or hundreds of millions of network packets transitingbetween an originating source and a final destination, passing throughmultiple network routers and switches in between. Network packets, ormessages, can be grouped together into “flows” based on commoncharacteristics including, but not limited to, source IP address,destination IP address, source port number, destination port number, andprotocol type.

To quickly identify problems and determine when and where the problemsoccurred, network operators must have the ability to visualize, inreal-time, the state of the network, including, but not limited to, thethroughput, loss, jitter, latency, errors, retransmits, andfragmentations of the network. In addition, it is paramount for networkoperators to understand and visualize the flows on the network todetermine the paths of the packets on the network in order to determinethe optimal distribution of packets.

FIG. 10 illustrates one embodiment of a network visualization display1000. The network visualization display 1000 provides visualization ofnetwork states, flows, and relationships between nodes of the network.The network visualization display 1000 comprises a chord diagram 1002.The chord diagram 1002 is a circular diagram with one or more nodes 1004on the edge of the circle. One or more chords 1006 represent therelationships, or flows, between the one or more nodes 1004. The chorddiagram 1002 is configured to provide visualization of a network inreal-time and “at-a-glance” to a network operator.

The chord diagram 1002 illustrates the relationship between the nodes1004. The nodes 1004 represent, for example, network elements, virtualnetwork elements, and/or networks of networks. The chord diagram 1002further illustrates the relationship between networks of interdependentnetworks, traffic flow and directions of flows between nodes 1004,and/or traffic flows and directions of flows between interdependentnetworks. In some embodiments, the chord diagram 1002 illustratesvarious network metrics, such as, for example, throughput, latency,jitter, loss, retransmission rate, error, and/or other network analyticsmeasured by the network monitoring and analytics systems and methods.The chord diagram 1002 identifies the comparative utilization of networkelements, whether those elements are close to capacity, and thecomparative amount of traffic flow between nodes. Errors, or problems,with the network and/or the relationship between errors, are identifiedin the chord diagram 1002. In some embodiments, historical error ratesand problems are identified on the chord diagram 1002. In variousembodiments, the network operations visualized by the chord diagram 1002are modifiable by, for example, dynamically changing the view byupdating network parameters, such as, for example, time duration, sortorder, selection of network elements, threshold of metrics, and/orgeographical areas.

In various embodiments, the nodes 1004 in the chord diagram 1002represent physical and/or virtual network elements, such as, forexample, routers, switches, firewalls, intrusion detection/preventiondevices, network monitoring/metering devices, multiple network elements,and/or networks of networks. The network visualization display 1000enables encoding of a large volume of information for each of the nodes1004, such as, for example, throughput, utilization ratio, thresholds,mean and/or average utilization. Throughput comprises the volume of dataflowing through a node 1004 and is measured, for example, in megabitsper second (Mbps). Utilization ratio comprises the current traffic loadat a node 1004 and/or over the network compared to the maximum trafficload that the individual node 1004 and/or overall network can handle andis presented, for example, as a percentage of maximum traffic load.Thresholds comprise user-defined thresholds of utilization and/orthroughput. In some embodiments, visual indicators are providedindicating whether the utilization and/or throughput has met theuser-defined threshold values. The mean/average utilization comprisesthe average traffic flow the node 1004 and/or network.

FIG. 11 illustrates a section 1050 of the chord diagram 1002. The chorddiagram 1002 comprises a plurality of nodes 1004. Visual elements may bepresented to the user to identify network problems. Network errorsand/or issues may by transient. A problem in the network may be maskedby a change in traffic flow and may not be apparent during a fixed timewithin the network. Transient problems may indicate larger issues withinthe network. The network visualization display 1000 provides a real-timevisual indication of the network to identify transient network issuesand address larger issues within the network.

In some embodiments, the network visualization display 1000 utilizesvisual indicators, such as, for example, colored elements, to identifyissues within the network. In the illustrated embodiment, the pluralityof nodes 1004 are presented as color bands. The color of the bandindicates the status of the node 1004. For example, in the illustratedembodiment, a first node 1004 a is illustrated as an orange band, asecond node 1004 b is illustrated as a red band, and a third node 1004 cis illustrated as a grey node. The orange color of the first node 1004 aindicates that a non-urgent issue exists at the first node 1004 a. Forexample, the first node 1004 a may be experiencing above average trafficor a slight degradation of performance. The red color of the second node1004 b indicates an urgent issue exists at the second node 1004 b. Forexample, the second node 1004 b may have gone down, may not be sendingtraffic, and/or may be heavily over-utilized and is dropping packets.The grey color of the third node 1004 c may indicate that the third node1004 c is operating within normal operating parameters and that noissues currently exist at the third node 1004 c.

In some embodiments, a visual indicator is provided to draw attention toone or more nodes 1004 b experiencing urgent issues requiring immediateattention. For example, in the illustrated embodiment, a dot 1008 isprovided as a visual indicator of a node that currently has and/or haspreviously had problems during a selected time period. The dot 1008enables network operators to identify nodes 1004 b that havehistorically and/or are currently experiencing issues. A large number ofdots 1008, or other visual indicators, indicate larger network problems.The dot 1008 comprises a color scheme similar to the node color bandsdiscussed above, and identifies urgent and non-urgent issues based onthe color of the dot.

In the illustrated embodiment, the center ring 1010 represents theaverage traffic utilization for each of the nodes 1004 during theselected time period. In some embodiments, the average trafficutilization is calculated over the entire life of the node 1004, for apredetermined time period, and/or for a user defined time period. Theactual value represented by the center ring 1010 may vary from node tonode but will always represent the average utilization for the selectedtime period of each of the nodes 1004.

In some embodiments, the length of the band representing each of thenodes 1004 is determined by the standard deviation of the node 1004 fromthe average utilization of the node 1004. A longer node 1004 in thepositive direction away from the center ring 1008 indicates an aboveaverage traffic utilization. A longer node 1004 in a negative directionfrom the center ring 1008 indicates below-average traffic utilizationfor the node 1004. Conversely, the shorter the node 1004 in eitherdirection, the lower the standard deviation of the traffic flow of thenode 1004 from the average traffic utilization. In some embodiments,three possible sizes are defined for each node: long, medium, and short.A long size is +/−2.5 standard deviations from the average, andrepresents about 1% of the nodes 1004 in the illustrated embodiment. Themedium size is +/−1.96 standard deviations from the average, and inrepresents about 5% of the 1004 nodes in the illustrated embodiment. Theshort size is +/−1 standard deviation from the average, and representsabout 31.7% of the nodes 1004 in the illustrated embodiment. Althoughspecific values have been given for long, medium, and short nodes, thoseskilled in the art will recognize that these values are given by way ofexample only, and are not intended to be limited. Any value of standarddeviation may be set for each of the long, medium, or short valuethresholds.

By visualizing the size of the nodes 1004 using standard deviation, anetwork operator can quickly, and at-a-glance, determine which of thenetwork nodes 1004 are reaching critical issues, such as, for example,nodes that are +2.5 standard deviations from average. Network operatorscan also leverage the visualization to identify network elements thathave very low utilization, such as, for example, nodes that are −2.5standard deviations from average. The network operator and/or anautomated system can rebalance the network traffic to take advantage ofunder utilized equipment instead of purchasing additional networkequipment.

In some embodiments, the direction of growth of each node 1004 away fromthe center ring 1008 indicates over and/or under utilization of the node1004 compared to the average utilization of the node 1004. An outwardgrowth of a node 1004 from the center ring 1008 indicates overutilization. An inward growth of a node 1004 from the center ring 1008indicates under utilization. The combination of length and direction ofthe node provides network operators a clear visualization of the currentutilization of network equipment. For example, in the illustratedembodiment, a second node 1004 b extends +2.5 standard deviations fromthe center ring 1010 indicating a critical level of above averagetraffic flow.

In some embodiments, the color of each node 1004 indicates the status ofthe node 1004. For example, in the illustrated embodiment, three colors,grey, orange, and red, indicate the status of various nodes. A grey nodeindicates a node 1004 within a normal deviation from the averageutilization of the node. An orange node indicates a non-urgent issuewith a node 1004 a. A red node indicates an urgent issue with a node1004 b. For example, a red node may indicate that a network element hasgone offline and/or has not been sending traffic or may indicate thatthe network element is heavily over-utilized and is dropping packets.The network visualization display 1000 enables full precisionvisualization of a network within a constrained space and provides fullinteractivity for network operators to enable operators to understandand/or explore the network.

In some embodiments, the network visualization display 1000 illustratesa global network comprising a plurality of interdependent localnetworks. Each node 1004 of the network visualization display 1000 mayrepresent one or more local networks. The chords 1006 represent therelationships and flows between the interdependent local networks. Insome embodiments, the network visualization display 1000 comprises oneor more layers, allowing a network operator to drilldown into a specificnode and retrieve a chord diagram 1002 representative of the localnetwork represented by the node 1004 in the overall networkvisualization display 1000. Based on the design of the network,additional layers may be present at each of the network layers. Forexample, in one embodiment, a distributed global network comprises threenetwork levels, a global level, a country level, and a regional level.The global level comprises a chord network comprising nodes representingnetworks within one or more countries, such as, for example, GreatBritain, the United States, Spain, Germany, and/or additional countries.A network operator may drilldown into each of the nodes of the globalnetwork chord network to view a second network level chord diagramrepresentative of a nationwide network within the selected country. Forexample, a network operator may drilldown into the node representativeof the United States and receive a chord diagram representative of theconnections between networks within major cities of the United States,such as, for example, New York, Los Angeles, San Francisco, Chicago,and/or other cities. The network operator may further drilldown intoeach of the city nodes of the nationwide chord diagram to view a chorddiagram representative of a local network within each of the specificcities. For example, a network operator may select the New York node toreceive a chord diagram representative of the local mobile networkwithin New York City.

In some embodiments, the network visualization display 1000 providesvisualization of network equipment utilization. A visual indicator 1008may identify over-utilization and under-utilization of networkequipment. Over-utilization of network equipment may indicatebottlenecks and may require the acquisition of additional networkequipment to handle the network load. Under, or low, utilization mayindicate over-provisioned or over-equipped networks. Over-provisionednetworks may prevent a company or organization from recouping investmentas quickly as predicted. The ability to visualize the utilization of anetwork provides mission-critical information to network operators.

FIG. 12 illustrates one embodiment of a chord diagram 1102 comprising aplurality of chords 1106 a-1106 c. The plurality of chords 1106 a-1106 crepresents the relationships amongst and between the network nodes 1104and illustrates the quantity of information flowing between the nodes1104. The plurality of chords 1106 a-1106 c provide network operatorsthe ability to explore the network and network flow to visualize, inreal-time, existing, prior, or developing network problems. The chords1106 a-1106 c comprise various features to provide information tonetwork operators, such as, for example, direction, origin, color,width, and path. The direction of the traffic flow represented by achord 1106 a-1106 c is indicated by an arrow 1110 at the end of thepath, which identifies the destination node 1104 b of the traffic flow.The originating node 1104 a is the node that does not have an arrowindicator.

Traffic flow of a single node 1104 a represents traffic originating atand/or terminating at the selected node 1104 a. In some embodiments, oneor more chords 1106 a-1106 c illustrate the traffic originating from aselected node 1104 a. The traffic originating at a selected node 1104 ais illustrated as one or more chords 1106 a-1106 c radiating, orfanning-out, from the selected node 1104 a. The arrows 1110 a-1110 cindicate the destination node for each chord 1106 a-1106 c. In someembodiments, one or more chords illustrate the flow of traffic destinedfor a selected node 1104 b. The source of the traffic will come from oneor more other nodes 1104 a and will indicate the flow of traffic intothe selected node 1104 b.

In some embodiments, the color of the chords 1106 a-1106 c indicatespotential problems with traffic flow between one or more nodes 1104a-1104 d. The colors of a chord 1106 a-1106 c indicate, for example,excessive loss of packets, large jitter, large latency, excessiveretransmissions, and/or additional network flow issues. In someembodiments, the width of the chord 1106 a-1106 c indicates one or moreselected parameters, such as, for example, the volume of traffic betweena first node 1104 a and a second node 1104 b, the amount of latency,jitter, loss, errors, and/or retransmissions between a first node 1104 aand a second node 1104 b, and/or other selected parameters.

The path of traffic flow from a first node to a second node need notcomprise a direct path. For example, in one embodiment, a path from afirst node to a second node may pass through a third node and a fourthnode. For example, a packet originating at a first node is transmittedfrom the first node to a third node. The third node retransmits thepacket to a fourth node. The fourth node retransmits the packet to thesecond node. The path of traffic from a first node to a second node maybe represented as a series of chords comprising directional arrows toindicate the flow of traffic from each of the source nodes, destinationnode, and any intermediate nodes.

The network visualization display 1000 enables a network operator tointeract with different components on the network and/or exploredifferent parts, or branches, of the network. In some embodiments thenetwork visualization display 1000 allows a user to interact with anode, such as, for example, by hovering a mouse cursor over the node.Interacting with a node 1004 modifies the network visualization display1000. For example, interacting with a node 1004 may limit the chords1006 displayed on a chord diagram 1002 to only that traffic flowing toand/or from the selected node 1004. Interacting with a node 1004 maydisplay metrics for the selected node, such as, for example,upstream/downstream traffic throughput, latency, jitter, loss, errors,and/or retransmissions. One or more network nodes 1004 may be selectedby, for example, filtering using name, geographic location, utilization,throughput, and/or other parameters.

In some embodiments, the network visualization display 1000 isconfigured to change the display in response to user interactions. Forexample, if a user interacts with an un-highlighted node, the selectednode is highlighted as well as any nodes that connect directly to theselected node. The path between the selected node and any connectednodes is displayed as a chord. Interacting with a selected node togglesthe direction of traffic flow to and from the node. Selecting a chordhighlights the selected chord, the originating node, and the destinationnode for the chord. If a user defines a subset of nodes, the subset ofnodes, as well as any nodes connecting directly to at least one of thesubset of nodes, are selected.

FIG. 13 illustrates one embodiment of a computing device 1100 which canbe used in one embodiment of the systems and methods for networkmonitoring and analytics. For the sake of clarity, the computing device1100 is shown and described here in the context of a single computingdevice. It is to be appreciated and understood, however, that any numberof suitably configured computing devices can be used to implement any ofthe described embodiments. For example, in at least some implementation,multiple communicatively linked computing devices are used. One or moreof these devices can be communicatively linked in any suitable way suchas via one or more networks (LANs), one or more wide area networks(WANs) or any combination thereof.

In this example, the computing device 1100 comprises one or moreprocessor circuits or processing units 1102, on or more memory circuitsand/or storage circuit component(s) 1104 and one or more input/output(I/O) circuit devices 1106. Additionally, the computing device 1100comprises a bus 1108 that allows the various circuit components anddevices to communicate with one another. The bus 1108 represents one ormore of any of several types of bus structures, including a memory busor local bus using any of a variety of bus architectures. The bus 1108may comprise wired and/or wireless buses.

The processing unit 1102 may be responsible for executing varioussoftware programs such as system programs, applications programs, and/ormodule to provide computing and processing operations for the computingdevice 1100. The processing unit 1102 may be responsible for performingvarious voice and data communications operations for the computingdevice 1100 such as transmitting and receiving voice and datainformation over one or more wired or wireless communication channels.Although the processing unit 1102 of the computing device 1100 includessingle processor architecture as shown, it may be appreciated that thecomputing device 1100 may use any suitable processor architecture and/orany suitable number of processors in accordance with the describedembodiments. In one embodiment, the processing unit 1100 may beimplemented using a single integrated processor.

The processing unit 1102 may be implemented as a host central processingunit (CPU) using any suitable processor circuit or logic device(circuit), such as a as a general purpose processor. The processing unit1102 also may be implemented as a chip multiprocessor (CMP), dedicatedprocessor, embedded processor, media processor, input/output (I/O)processor, co-processor, microprocessor, controller, microcontroller,application specific integrated circuit (ASIC), field programmable gatearray (FPGA), programmable logic device (PLD), or other processingdevice in accordance with the described embodiments.

As shown, the processing unit 1102 may be coupled to the memory and/orstorage component(s) 1104 through the bus 1108. The memory bus 1108 maycomprise any suitable interface and/or bus architecture for allowing theprocessing unit 1102 to access the memory and/or storage component(s)1104. Although the memory and/or storage component(s) 1104 may be shownas being separate from the processing unit 1102 for purposes ofillustration, it is worthy to note that in various embodiments someportion or the entire memory and/or storage component(s) 1104 may beincluded on the same integrated circuit as the processing unit 1102.Alternatively, some portion or the entire memory and/or storagecomponent(s) 1104 may be disposed on an integrated circuit or othermedium (e.g., hard disk drive) external to the integrated circuit of theprocessing unit 1102. In various embodiments, the computing device 1100may comprise an expansion slot to support a multimedia and/or memorycard, for example.

The memory and/or storage component(s) 1104 represent one or morecomputer-readable media. The memory and/or storage component(s) 1104 maybe implemented using any computer-readable media capable of storing datasuch as volatile or non-volatile memory, removable or non-removablememory, erasable or non-erasable memory, writeable or re-writeablememory, and so forth. The memory and/or storage component(s) 1104 maycomprise volatile media (e.g., random access memory (RAM)) and/ornonvolatile media (e.g., read only memory (ROM), Flash memory, opticaldisks, magnetic disks and the like). The memory and/or storagecomponent(s) 1104 may comprise fixed media (e.g., RAM, ROM, a fixed harddrive, etc.) as well as removable media (e.g., a Flash memory drive, aremovable hard drive, an optical disk, etc.). Examples ofcomputer-readable storage media may include, without limitation, RAM,dynamic RAM (DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM(SDRAM), static RAM (SRAM), read-only memory (ROM), programmable ROM(PROM), erasable programmable ROM (EPROM), electrically erasableprogrammable ROM (EEPROM), flash memory (e.g., NOR or NAND flashmemory), content addressable memory (CAM), polymer memory (e.g.,ferroelectric polymer memory), phase-change memory, ovonic memory,ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS)memory, magnetic or optical cards, or any other type of media suitablefor storing information.

The one or more I/O devices 1106 allow a user to enter commands andinformation to the computing device 1100, and also allow information tobe presented to the user and/or other components or devices. Examples ofinput devices include a keyboard, a cursor control device (e.g., amouse), a microphone, a scanner and the like. Examples of output devicesinclude a display device (e.g., a monitor or projector, speakers, aprinter, a network card, etc.). The computing device 1100 may comprisean alphanumeric keypad coupled to the processing unit 1102. The keypadmay comprise, for example, a QWERTY key layout and an integrated numberdial pad. The computing device 1100 may comprise a display coupled tothe processing unit 1102. The display may comprise any suitable visualinterface for displaying content to a user of the computing device 1100.In one embodiment, for example, the display may be implemented by aliquid crystal display (LCD) such as a touch-sensitive color (e.g.,76-bit color) thin-film transistor (TFT) LCD screen. The touch-sensitiveLCD may be used with a stylus and/or a handwriting recognizer program.

The processing unit 1102 may be arranged to provide processing orcomputing resources to the computing device 1100. For example, theprocessing unit 1102 may be responsible for executing various softwareprograms including system programs such as operating system (OS) andapplication programs. System programs generally may assist in therunning of the computing device 1100 and may be directly responsible forcontrolling, integrating, and managing the individual hardwarecomponents of the computer system. The OS may be implemented, forexample, as a Microsoft® Windows OS, Symbian OS™, Embedix OS, Linux OS,Binary Run-time Environment for Wireless (BREW) OS, JavaOS, Android OS,Apple OS or other suitable OS in accordance with the describedembodiments. The computing device 1100 may comprise other systemprograms such as device drivers, programming tools, utility programs,software libraries, application programming interfaces (APIs), and soforth.

The computer 1100 also includes a network interface 1110 coupled to thebus 1108. The network interface 1110 provides a two-way datacommunication coupling to a local network 1112. For example, the networkinterface 1110 may be a digital subscriber line (DSL) modem, satellitedish, an integrated services digital network (ISDN) card or other datacommunication connection to a corresponding type of telephone line. Asanother example, the communication interface 1110 may be a local areanetwork (LAN) card effecting a data communication connection to acompatible LAN. Wireless communication means such as internal orexternal wireless modems may also be implemented.

In any such implementation, the network interface 1110 sends andreceives electrical, electromagnetic or optical signals that carrydigital data streams representing various types of information, such asthe selection of goods to be purchased, the information for payment ofthe purchase, or the address for delivery of the goods. The networkinterface 1110 typically provides data communication through one or morenetworks to other data devices. For example, the network interface 1110may effect a connection through the local network to an Internet HostProvider (ISP) or to data equipment operated by an ISP. The ISP in turnprovides data communication services through the internet (or otherpacket-based wide area network). The local network and the internet bothuse electrical, electromagnetic or optical signals that carry digitaldata streams. The signals through the various networks and the signalson the network interface 1110, which carry the digital data to and fromthe computer system 110, are exemplary forms of carrier wavestransporting the information.

The computer 1100 can send messages and receive data, including programcode, through the network(s) and the network interface 1110. In theInternet example, a server might transmit a requested code for anapplication program through the internet, the ISP, the local network(the network 1112) and the network interface 1110. In accordance withthe invention, one such downloaded application provides for theidentification and analysis of a prospect pool and analysis of marketingmetrics. The received code may be executed by processor 1104 as it isreceived, and/or stored in storage device 1110, or other non-volatilestorage for later execution. In this manner, computer 1100 may obtainapplication code in the form of a carrier wave.

Various embodiments may be described herein in the general context ofcomputer executable instructions, such as software, program modules,and/or engines being executed by a computer. Generally, software,program modules, and/or engines include any software element arranged toperform particular operations or implement particular abstract datatypes. Software, program modules, and/or engines can include routines,programs, objects, components, data structures and the like that performparticular tasks or implement particular abstract data types. Animplementation of the software, program modules, and/or enginescomponents and techniques may be stored on and/or transmitted acrosssome form of computer-readable media. In this regard, computer-readablemedia can be any available medium or media useable to store informationand accessible by a computing device. Some embodiments also may bepracticed in distributed computing environments where operations areperformed by one or more remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, software, program modules, and/or engines may be located inboth local and remote computer storage media including memory storagedevices.

Although some embodiments may be illustrated and described as comprisingfunctional components, software, engines, and/or modules performingvarious operations, it can be appreciated that such components ormodules may be implemented by one or more hardware components, softwarecomponents, and/or combination thereof. The functional components,software, engines, and/or modules may be implemented, for example, bylogic (e.g., instructions, data, and/or code) to be executed by a logicdevice (e.g., processor). Such logic may be stored internally orexternally to a logic device on one or more types of computer-readablestorage media. In other embodiments, the functional components such assoftware, engines, and/or modules may be implemented by hardwareelements that may include processors, microprocessors, circuits, circuitelements (e.g., transistors, resistors, capacitors, inductors, and soforth), integrated circuits, application specific integrated circuits(ASIC), programmable logic devices (PLD), digital signal processors(DSP), field programmable gate array (FPGA), logic gates, registers,semiconductor device, chips, microchips, chip sets, and so forth.

Examples of software, engines, and/or modules may include softwarecomponents, programs, applications, computer programs, applicationprograms, system programs, machine programs, operating system software,middleware, firmware, software modules, routines, subroutines,functions, methods, procedures, software interfaces, application programinterfaces (API), instruction sets, computing code, computer code, codesegments, computer code segments, words, values, symbols, or anycombination thereof. Determining whether an embodiment is implementedusing hardware elements and/or software elements may vary in accordancewith any number of factors, such as desired computational rate, powerlevels, heat tolerances, processing cycle budget, input data rates,output data rates, memory resources, data bus speeds and other design orperformance constraints.

In some cases, various embodiments may be implemented as an article ofmanufacture. The article of manufacture may include a computer readablestorage medium arranged to store logic, instructions and/or data forperforming various operations of one or more embodiments. In variousembodiments, for example, the article of manufacture may comprise amagnetic disk, optical disk, flash memory or firmware containingcomputer program instructions suitable for execution by a generalpurpose processor or application specific processor. The embodiments,however, are not limited in this context.

While various details have been set forth in the foregoing description,it will be appreciated that the various embodiments of the apparatus,system, and method for anonymous sharing and public vetting of contentmay be practiced without these specific details. For example, forconciseness and clarity selected aspects have been shown in blockdiagram form rather than in detail. Some portions of the detaileddescriptions provided herein may be presented in terms of instructionsthat operate on data that is stored in a computer memory. Suchdescriptions and representations are used by those skilled in the art todescribe and convey the substance of their work to others skilled in theart. In general, an algorithm refers to a self-consistent sequence ofsteps leading to a desired result, where a “step” refers to amanipulation of physical quantities which may, though need notnecessarily, take the form of electrical or magnetic signals capable ofbeing stored, transferred, combined, compared, and otherwisemanipulated. It is common usage to refer to these signals as bits,values, elements, symbols, characters, terms, numbers, or the like.These and similar terms may be associated with the appropriate physicalquantities and are merely convenient labels applied to these quantities.

Unless specifically stated otherwise as apparent from the foregoingdiscussion, it is appreciated that, throughout the foregoingdescription, discussions using terms such as “processing” or “computing”or “calculating” or “determining” or “displaying” or the like, refer tothe action and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

It is worthy to note that any reference to “one aspect,” “an aspect,”“one embodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the aspect isincluded in at least one aspect. Thus, appearances of the phrases “inone aspect,” “in an aspect,” “in one embodiment,” or “in an embodiment”in various places throughout the specification are not necessarily allreferring to the same aspect. Furthermore, the particular features,structures or characteristics may be combined in any suitable manner inone or more aspects.

Although various embodiments have been described herein, manymodifications, variations, substitutions, changes, and equivalents tothose embodiments may be implemented and will occur to those skilled inthe art. Also, where materials are disclosed for certain components,other materials may be used. It is therefore to be understood that theforegoing description and the appended claims are intended to cover allsuch modifications and variations as falling within the scope of thedisclosed embodiments. The following claims are intended to cover allsuch modification and variations.

In summary, numerous benefits have been described which result fromemploying the concepts described herein. The foregoing description ofthe one or more embodiments has been presented for purposes ofillustration and description. It is not intended to be exhaustive orlimiting to the precise form disclosed. Modifications or variations arepossible in light of the above teachings. The one or more embodimentswere chosen and described in order to illustrate principles andpractical application to thereby enable one of ordinary skill in the artto utilize the various embodiments and with various modifications as aresuited to the particular use contemplated. It is intended that theclaims submitted herewith define the overall scope.

Some or all of the embodiments described herein may generally comprisetechnologies which can be implemented, individually, and/orcollectively, by a wide range of hardware, software, firmware, or anycombination thereof can be viewed as being composed of various types of“electrical circuitry.” Consequently, as used herein “electricalcircuitry” includes, but is not limited to, electrical circuitry havingat least one discrete electrical circuit, electrical circuitry having atleast one integrated circuit, electrical circuitry having at least oneapplication specific integrated circuit, electrical circuitry forming ageneral purpose computing device configured by a computer program (e.g.,a general purpose computer configured by a computer program which atleast partially carries out processes and/or devices described herein,or a microprocessor configured by a computer program which at leastpartially carries out processes and/or devices described herein),electrical circuitry forming a memory device (e.g., forms of randomaccess memory), and/or electrical circuitry forming a communicationsdevice (e.g., a modem, communications switch, or optical-electricalequipment). Those having skill in the art will recognize that thesubject matter described herein may be implemented in an analog ordigital fashion or some combination thereof.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a floppy disk, a hard disk drive, a Compact Disc (CD), aDigital Video Disk (DVD), a digital tape, a computer memory, etc.; and atransmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link (e.g., transmitter,receiver, transmission logic, reception logic, etc.), etc.).

One skilled in the art will recognize that the herein describedcomponents (e.g., operations), devices, objects, and the discussionaccompanying them are used as examples for the sake of conceptualclarity and that various configuration modifications are contemplated.Consequently, as used herein, the specific exemplars set forth and theaccompanying discussion are intended to be representative of their moregeneral classes. In general, use of any specific exemplar is intended tobe representative of its class, and the non-inclusion of specificcomponents (e.g., operations), devices, and objects should not be takenlimiting.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations are not expressly set forth herein for sakeof clarity.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures may beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable,” to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents, and/or wirelessly interactable, and/or wirelesslyinteracting components, and/or logically interacting, and/or logicallyinteractable components.

In some instances, one or more components may be referred to herein as“configured to,” “configurable to,” “operable/operative to,”“adapted/adaptable,” “able to,” “conformable/conformed to,” etc. Thoseskilled in the art will recognize that “configured to” can generallyencompass active-state components and/or inactive-state componentsand/or standby-state components, unless context requires otherwise.

While particular aspects of the present subject matter described hereinhave been shown and described, it will be apparent to those skilled inthe art that, based upon the teachings herein, changes and modificationsmay be made without departing from the subject matter described hereinand its broader aspects and, therefore, the appended claims are toencompass within their scope all such changes and modifications as arewithin the true spirit and scope of the subject matter described herein.It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to claims containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, those skilled in the art will recognize that suchrecitation should typically be interpreted to mean at least the recitednumber (e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that typically a disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms unless context dictates otherwise. For example, the phrase “Aor B” will be typically understood to include the possibilities of “A”or “B” or “A and B.”

With respect to the appended claims, those skilled in the art willappreciate that recited operations therein may generally be performed inany order. Also, although various operational flows are presented in asequence(s), it should be understood that the various operations may beperformed in other orders than those which are illustrated, or may beperformed concurrently. Examples of such alternate orderings may includeoverlapping, interleaved, interrupted, reordered, incremental,preparatory, supplemental, simultaneous, reverse, or other variantorderings, unless context dictates otherwise. Furthermore, terms like“responsive to,” “related to,” or other past-tense adjectives aregenerally not intended to exclude such variants, unless context dictatesotherwise.

Those skilled in the art will recognize that it is common within the artto implement devices and/or processes and/or systems, and thereafter useengineering and/or other practices to integrate such implemented devicesand/or processes and/or systems into more comprehensive devices and/orprocesses and/or systems. That is, at least a portion of the devicesand/or processes and/or systems described herein can be integrated intoother devices and/or processes and/or systems via a reasonable amount ofexperimentation. Those having skill in the art will recognize thatexamples of such other devices and/or processes and/or systems mightinclude—as appropriate to context and application—all or part of devicesand/or processes and/or systems of (a) an air conveyance (e.g., anairplane, rocket, helicopter, etc.), (b) a ground conveyance (e.g., acar, truck, locomotive, tank, armored personnel carrier, etc.), (c) abuilding (e.g., a home, warehouse, office, etc.), (d) an appliance(e.g., a refrigerator, a washing machine, a dryer, etc.), (e) acommunications system (e.g., a networked system, a telephone system, aVoice over IP system, etc.), (f) a business entity (e.g., an InternetService Provider (ISP) entity such as Comcast Cable, Qwest, SouthwesternBell, etc.), or (g) a wired/wireless services entity (e.g., Sprint,Cingular, Nextel, etc.), etc.

In certain cases, use of a system or method may occur in a territoryeven if components are located outside the territory. For example, in adistributed computing context, use of a distributed computing system mayoccur in a territory even though parts of the system may be locatedoutside of the territory (e.g., relay, server, processor, signal-bearingmedium, transmitting computer, receiving computer, etc. located outsidethe territory).

A sale of a system or method may likewise occur in a territory even ifcomponents of the system or method are located and/or used outside theterritory. Further, implementation of at least part of a system forperforming a method in one territory does not preclude use of the systemin another territory.

In summary, numerous benefits have been described which result fromemploying the concepts described herein. The foregoing description ofthe one or more embodiments has been presented for purposes ofillustration and description. It is not intended to be exhaustive orlimiting to the precise form disclosed. Modifications or variations arepossible in light of the above teachings. The one or more embodimentswere chosen and described in order to illustrate principles andpractical application to thereby enable one of ordinary skill in the artto utilize the various embodiments and with various modifications as aresuited to the particular use contemplated. It is intended that theclaims submitted herewith define the overall scope.

Various aspects of the subject matter described herein are set out inthe following numbered clauses:

1. A computer-implemented method, comprising: calculating, by aprocessor, a sparse hash value for a first message at a plurality ofobservation points on a network using a first sparse hash function;associating, by the processor, metadata with the sparse hash value ofthe first message; tracking, by the processor, the transit of the firstmessage over the network; and generating, by the processor, one or morenetwork analytics for the first message over the network, wherein theone or more network analytics are generated from the associatedmetadata.

2. The computer-implemented method of clause 1, comprising calculating,by the processor, the sparse hash value of the first message for one ormore invariant fields of the first message.

3. The computer-implemented method of clause 2, comprising treating, bythe processor, one or more variant fields as a constant value duringcalculation of the sparse hash value.

4. The computer-implemented method of clause 2, comprising: calculating,by the processor, a sparse hash value for each of a plurality ofmessages at the plurality of observation points on the network using thefirst sparse hash function; associating, by the processor, metadata withthe has value calculated for each of the plurality of messages;tracking, by the processor, the transit of the plurality of messagesover the network; and generating, by the processor, the one or morenetwork analytics for the plurality of messages over the network.

6. The computer-implemented method of clause 4, wherein the first sparsehash function generates sparse hash values that are highly probable tobe unique for the first plurality of messages.

7. The computer-implemented method of clause 4, comprising: calculating,by the processor, a sparse hash value for a second plurality of messagesat the plurality of observation points on the network using a secondsparse hash function, wherein the first sparse hash function and thesecond sparse hash function are different.

8. The computer-implemented method of clause 1, wherein the one or morenetwork analytics comprises at least one of throughput, loss, jitter,latency, errors, retransmits, and fragmentation of packets.

9. An apparatus comprising: a processor; and a non-transitorycomputer-readable medium coupled to the processor, the non-transitorycomputer-readable medium configured to store computer programinstructions that when executed by the processor are operable to causethe processor to: calculate a sparse hash value for a first message at aplurality of observation points on a network using a first sparse hashfunction; associate metadata with the sparse hash value calculated forthe first message; track, the transit of the first message over thenetwork; and generate one or more network analytics for the firstmessage over the network.

10. The apparatus of clause 9, wherein the processor is furtherconfigured to calculate the sparse hash value of the first message usingone or more invariant fields of the first message.

11. The apparatus of clause 10, wherein the processor is furtherconfigured to treat one or more variant fields of the first message as aconstant value during calculation of the sparse hash value.

12. The apparatus of clause 10, wherein the processor is furtherconfigured to: calculate a sparse hash value for each of a plurality ofmessages at the plurality of observation points on the network using thefirst sparse hash function; associate metadata with the sparse hashvalue calculated for each of the plurality of messages; track thetransit of the plurality of messages over the network; and generate theone or more network analytics for the plurality of messages over thenetwork.

13. The apparatus of clause 12, wherein the first sparse hash functiongenerates sparse hash values that are highly probable to be unique forthe first plurality of messages.

14. The apparatus of clause 12, wherein the processor is furtherconfigured to calculate a sparse hash value for a second plurality ofmessages at the plurality of observation points on the network using asecond sparse hash function, wherein the first sparse hash function andthe second sparse hash function are different.

15. The apparatus of clause 9, wherein the one or more network analyticscomprises at least one of throughput, loss, jitter, latency, errors,retransmits, and fragmentation of packets.

16. A computer-implemented method comprising: receiving, by a processor,a plurality of metadata packets corresponding to a plurality ofmessages, wherein each of the metadata packets comprises a sparse hashvalue; identifying, by the processor, a plurality of matching sparsehash values, wherein the plurality of matching sparse hash valuescorrespond to at least a first message and a second message;disambiguating, by the processor, the first message and the secondmessage, wherein the first message and the second message aredisambiguated using the metadata associated with the plurality of sparsehash values.

17. The computer-implemented method of clause 16, comprising:determining, by the processor, a first existence window for the firstmessage; comparing, by the processor, a time of arrival of each of theplurality of metadata packets to the first existence window; andidentifying, by the processor, a subset of the plurality of metadatapackets corresponding to the first message by identifying one or moremetadata packets comprising matching sparse hash values and a time ofarrival within the first existence window for the first message.

18. The computer-implemented method of clause 16, comprising:determining, by the processor, a first trajectory for the first message;comparing, by the processor, a location field for each of the pluralityof metadata packets to the first trajectory for the first message; andidentifying, by the processor, a subset of the plurality of metadatapackets corresponding to the first message by identifying one or moremetadata packets comprising matching sparse hash values and the locationfield corresponding to the first trajectory.

19. The computer-implemented method of clause 16, comprising: comparing,by the processor, each of the plurality of metadata packets;identifying, by the processor, the first message by identifying a firstsubset of the plurality of metadata packets comprising matching sparsehash values and at least one invariant field of the associated message,wherein the invariant field is used in the calculation of the sparsehash value; and identifying, by the processor, the second message byidentifying a second subset of the plurality of metadata packetscomprising matching sparse hash values and at least one invariant fieldof the associated message, wherein the invariant field is used in thecalculation of the sparse hash value.

20. The computer-implemented method of clause 19, wherein the metadataassociated with the plurality of matching sparse hash values comprises alength of the message corresponding to the sparse hash value.

21. The computer-implemented method of clause 19, wherein the metadataassociated with the plurality of matching sparse hash values comprises asubset of the invariant field.

22. The computer-implemented method of clause 16, wherein each of theplurality of metadata packets comprise comprises aggregated metadata,wherein the aggregated metadata comprises metadata for at least two ofthe observation points through which the associated message hastraveled.

23. An apparatus comprising: a processor; and a non-transitorycomputer-readable medium coupled to the processor, the non-transitorycomputer-readable medium configured to store computer programinstructions that when executed by the processor are operable to causethe processor to: receive a plurality of metadata packets correspondingto a plurality of messages, wherein each of the metadata packetscomprises a sparse hash value; identify a plurality of matching sparsehash values, wherein the plurality of matching sparse hash valuescorrespond to at least a first message and a second message;disambiguate the first message and the second message, wherein the firstmessage and the second message are disambiguated using the metadataassociated with the plurality of sparse hash values.

24. The apparatus of clause 23, wherein the processor is furtherconfigured to: determine a first existence window for the first message;compare a time of arrival of each of the plurality of metadata packetsto the first existence window; and identify a subset of the plurality ofmetadata packets corresponding to the first message by identifying oneor more metadata packets comprising matching sparse hash values and atime of arrival within the first existence window for the first message.

25. The apparatus of clause 23, wherein the processor is furtherconfigured to: determine a first trajectory for the first message;compare a location field for each of the plurality of metadata packetsto the first trajectory for the first message; and identify a subset ofthe plurality of metadata packets corresponding to the first message byidentifying one or more metadata packets comprising matching sparse hashvalues and the location field corresponding to the first trajectory.

26. The apparatus of clause 23, wherein the processor is furtherconfigured to: compare each of the plurality of metadata packets;identify the first message by identifying a first subset of theplurality of metadata packets comprising matching sparse hash values andat least one invariant field of the associated message, wherein theinvariant field is used in the calculation of the sparse hash value; andidentify the second message by identifying a second subset of theplurality of metadata packets comprising matching sparse hash values andat least one invariant field of the associated message, wherein theinvariant field is used in the calculation of the sparse hash value.

27. The apparatus of clause 26, wherein the metadata associated with theplurality of matching sparse hash values comprises a length of themessage corresponding to the sparse hash value.

28. The apparatus of clause 26, wherein the metadata associated with theplurality of matching sparse hash values comprises a subset of theinvariant field.

29. The apparatus of clause 26, wherein each of the plurality ofmetadata packets comprise comprises aggregated metadata, wherein theaggregated metadata comprises metadata for each observation pointthrough which the associated message has traveled.

30. A computer-implemented method, comprising: generating, by aprocessor, a network visualization display to provide a visualindication of one or more network analytics, wherein the networkvisualization display comprises: a plurality of nodes; and one or moreconnections between the plurality of nodes.

31. The computer-implemented method of clause 30, wherein the networkvisualization display comprises a chord diagram, wherein the pluralityof nodes are arranged in a circular configuration, and wherein the oneor more connections between the plurality of nodes are displayed aschords.

32. The computer-implemented method of clause 31, comprising:generating, by the processor, a visual representation of each of theplurality of nodes, wherein the visual representation is indicative of atraffic flow of the node.

33. The computer-implemented method of clause 32, wherein the visualrepresentation comprises a distance from a center line, wherein thecenter line is representative of an average traffic flow of the node,and wherein the distance from the center line is indicative of thetraffic flow of the node compared to the average traffic flow of thenode.

34. The computer-implemented method of clause 31, wherein each of theplurality of nodes represents a network, and wherein the chord diagramrepresents a network of networks.

35. The computer-implemented method of clause 31, wherein each of theplurality of nodes represents a network device, and wherein the chorddiagram represents a local network.

36. The computer-implemented method of clause 31, comprising generating,by the processor, a visual indicator indicative of a mission-criticalproblem at a node.

37. The computer-implemented method of clause 31, comprising generating,by the processor, a color for each of the plurality of nodes, whereinthe color indicates the status of the node.

38. The computer-implemented method of clause 31, comprising: receiving,by the processor, a selection of a first node; generating, by theprocessor, a traffic flow of the first node; and generating, by theprocessor, one or more chords indicative of the traffic flow between thefirst node and one or more additional nodes.

39. An apparatus comprising: a processor; and a non-transitorycomputer-readable medium coupled to the processor, the non-transitorycomputer-readable medium configured to store computer programinstructions that when executed by the processor are operable to causethe processor to: generate a network visualization display to provide avisual indication of one or more network analytics, wherein the networkvisualization display comprises: a plurality of nodes; and one or moreconnections between the plurality of nodes.

40. The apparatus of clause 39, wherein the network visualizationdisplay comprises a chord diagram, wherein the plurality of nodes arearranged in a circular configuration, and wherein the one or moreconnections between the plurality of nodes are displayed as chords.

41. The apparatus of clause 40, wherein the processor is furtherconfigured to generate a visual representation of each of the pluralityof nodes, wherein the visual representation is indicative of a trafficflow of the node.

42. The apparatus of clause 41, wherein the visual representationcomprises a distance from a center line, wherein the center line isrepresentative of an average traffic flow of the node, and wherein thedistance from the center line is indicative of the traffic flow of thenode compared to the average traffic flow of the node.

43. The apparatus of clause 40, wherein each of the plurality of nodesrepresents a network, and wherein the chord diagram represents a networkof networks.

44. The apparatus of clause 40, wherein each of the plurality of nodesrepresents a network device, and wherein the chord diagram represents alocal network.

45. The apparatus of clause 40, wherein the processor is furtherconfigured to generate a visual indicator indicative of amission-critical problem at a node.

46. The apparatus of clause 40, wherein the processor is furtherconfigured to generate a color for each of the plurality of nodes,wherein the color indicates the status of the node.

47. The apparatus of clause 40, wherein the processor is furtherconfigured to: receive a selection of a first node; generate a trafficflow of the first node; and generate one or more chords indicative ofthe traffic flow between the first node and one or more additionalnodes.

1. A computer-implemented method, comprising: calculating, by aprocessor, a sparse hash value for a first message at a plurality ofobservation points on a network using a first sparse hash function;associating, by the processor, metadata with the sparse hash value ofthe first message; tracking, by the processor, transit of the firstmessage over the network; and generating, by the processor, one or morenetwork analytics for the first message over the network, wherein theone or more network analytics are generated from the associatedmetadata.
 2. The computer-implemented method of claim 1, furthercomprising calculating, by the processor, the sparse hash value of thefirst message for one or more invariant fields of the first message. 3.The computer-implemented method of claim 2, further comprising treating,by the processor, one or more variant fields as a constant value duringcalculation of the sparse hash value.
 4. The computer-implemented methodof claim 1, further comprising: calculating, by the processor, a sparsehash value for each of a plurality of messages at the plurality ofobservation points on the network using the first sparse hash function;associating, by the processor, metadata with the hash value calculatedfor each of the plurality of messages; tracking, by the processor, thetransit of the plurality of messages over the network; and generating,by the processor, the one or more network analytics for the plurality ofmessages over the network.
 5. The computer-implemented method of claim4, wherein the first sparse hash function generates sparse hash valuesthat are highly probable to be unique for the first plurality ofmessages.
 6. The computer-implemented method of claim 4, furthercomprising: calculating, by the processor, a sparse hash value for asecond plurality of messages at the plurality of observation points onthe network using a second sparse hash function, wherein the firstsparse hash function and the second sparse hash function are different.7. The computer-implemented method of claim 1, wherein the one or morenetwork analytics comprises at least one of throughput, loss, jitter,latency, errors, retransmits, and fragmentation of packets.
 8. Thecomputer-implemented method of claim 1, further comprising mapping oneor more variant fields of the first message to an invariant value thatis used in the first sparse hash function.
 9. The computer-implementedmethod of claim 1, wherein the associated metadata comprises at leastone of a size of the first message, a time of transmission of the firstmessage, a trajectory of the first message through the network, a sourceof the first message, a destination of the first message, and a type ofpacket of the first message.
 10. The computer-implemented method ofclaim 1, further comprising determining an existence and a location oforiginal fields of the first message.
 11. An apparatus comprising: aprocessor; and a non-transitory computer-readable medium coupled to theprocessor, the non-transitory computer-readable medium configured tostore computer program instructions that when executed by the processorare operable to cause the processor to: calculate a sparse hash valuefor a first message at a plurality of observation points on a networkusing a first sparse hash function; associate metadata with the sparsehash value calculated for the first message; track, the transit of thefirst message over the network; and generate one or more networkanalytics for the first message over the network.
 12. The apparatus ofclaim 11, wherein the processor is further configured to calculate thesparse hash value of the first message using one or more invariantfields of the first message.
 13. The apparatus of claim 12, wherein theprocessor is further configured to treat one or more variant fields ofthe first message as a constant value during calculation of the sparsehash value.
 14. The apparatus of claim 11, wherein the processor isfurther configured to: calculate a sparse hash value for each of aplurality of messages at the plurality of observation points on thenetwork using the first sparse hash function; associate metadata withthe sparse hash value calculated for each of the plurality of messages;track the transit of the plurality of messages over the network; andgenerate the one or more network analytics for the plurality of messagesover the network.
 15. The apparatus of claim 14, wherein the firstsparse hash function generates sparse hash values that are highlyprobable to be unique for the first plurality of messages.
 16. Theapparatus of claim 14, wherein the processor is further configured tocalculate a sparse hash value for a second plurality of messages at theplurality of observation points on the network using a second sparsehash function, wherein the first sparse hash function and the secondsparse hash function are different.
 17. The apparatus of claim 11,wherein the one or more network analytics comprises at least one ofthroughput, loss, jitter, latency, errors, retransmits, andfragmentation of packets.
 18. The apparatus of claim 11, wherein theprocessor is further configured to map one or more variant fields of thefirst message to an invariant value that is used in the first sparsehash function.
 19. The apparatus of claim 11, wherein the associatedmetadata comprises at least one of a size of the first message, a timeof transmission of the first message, a trajectory of the first messagethrough the network, a source of the first message, a destination of thefirst message, and a type of packet of the first message.
 20. Theapparatus of claim 11, wherein the processor is further configured todetermine an existence and a location of original fields of the firstmessage.